Playing Card Data

In the user study, we chose to use playing card data for several reasons:

During the teaching session of the user study, participants are asked to classifify four classes: the aces, the nines, the queens and the kings. We provide them extra cards, the jacks, the twos and the tens and to be able to try out outliers.

Data collection

We collected 600 playing card images with a camera mounted on a stand shown on the following figure:

We only used the cards participants will have access to. It includes:

The data was processed using a script available in src/utilities.py file. The processed images are stored in a pickle file accessible online. The following cells download and import the data into the notebook.

Let's have a look to the training, testing and uncertain data.

Data annotation

The main author annotated each image with a label about the perceived uncertainty:

We used a labeling tool in the jupyter notebook as follow:

We also label the training and test data as "Classifiable"

Data analysis in the feature space

To speed up the training and enable interactive and incremental training during the user study, we had to perform transfer learning. To do so, we investigated embeddings that transform our data into a feature space.

We investigate four different embeedings:

To better estimate the richness and the variance distribution within our four feature spaces, we perform Principal Component Analysis (PCA).

We compute the Shannon entropy using the 10 first eigen-values of the Principal Analysis decomposition.

Observation: The variance in the data is more or less distributed across the components:

The Shannon entropy on the 10 first eigen values of the PCA is a good indicator of the variance distribution among the components. We might expect better separation between uncertain and certain data with ResNet50 since the variance is more distributed across several dimension.

We might expect better separation of the classes or the ambiguous or novel inputs with MobileNetV2 since the variance is more distributed across several dimension.

Let's vizualize the feature space for these two embeedings.

Observations: Visually, we observe a more coherent separation of the different types of card with the MobileNetV2 embedding and other cluster might exist in the remaining components. This confirms the intuition that we might expect a better separation of the data with MobileNetV2 embedding.

The introduction of the dataset, its collection, processing, annotation and visualization in different feature spaces is now over. Please go to the next notebook to learn about the uncertainty estimation used in the benchmark: 2_uncertainty_estimations.ipynb or return to the index